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Abstract — This paper proposes a statistical method for the 
analysis of multiple responses or outcome data in case control 
studies including situations in which the observations are either 
continuous or frequency data. Test statistics are proposed for 
assessing the statistical significance of differences between 
case-control response score. The proposed methods are 
illustrated with some sample data. When there only three 
possible response options in which the proposed method and the 
Stuart- Maxwell test can be equally used to analyse the data, the 
proposed test statistic is show to be at least as powerful as the 
Stuart-Maxwell test statistic. 

Index Terms — Multiple Response, Case Control, Scores, Test 
Statistic, Treatment, Prospective, Retrospective 

I. INTRODUCTION 

Often in controlled comparative prospective or retrospective 
studies involving matched samples of subjects or patients, the 
response of a subject to a predisposing factor in a 
retrospective study or to a condition or treatment in a 
prospective study may be dichotomous with only two possible 
naturally exclusive outcomes and appropriate for analysis 
using the McNemar Test (Gibbonsl973). But the responses 
may be much finer than simply dichotomous, assuming 
several possible values. For example in a retrospective study 
where the predisposing factor may be a subject’s employment 
status, a subject may be classified as unemployed, self 
employed, public servant, student, housewife etc. In a 
prospective study involving some conditions or tests, subjects 
or patients may be classified as recovered, much improved, 
improved, no change, worse or dead. A treatment or drug may 
be graded as very effective, effective, ineffective etc 
If there are only three possible response options or categories, 
then the Stuart-Maxwell test (Fleiss, 1981; Robertson et al, 
1974; Schlesselman, 1992; Zhao and Kolonel, 1992; Box and 
Cox, 1964; Maxwell, 1970; Stuart, 1955; Fleiss, 1981; 
Everitt, 1977) may be used to analyse the data. We here 
propose an alternative and easier to use method that is often 
more powerful than the usual Stuart/Maxwell test for three 
outcomes in a clinical trial and which is easily generalisable 
when there are more than three outcomes. 


II. The Proposed Method 

Suppose we have a random sample of n pairs of patients or 
subjects matched on a number of characteristics to be exposed 
to two experimental conditions, treatments, drugs or tests. 
Suppose further that the responses of these pairs of subjects 

are more than dichotomous but numbering c[ c > 3) 
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possible response options. Suppose further that the i th pair of 

patients is selected, for i =1 , 2 , n and one member 

of the pair is randomly assigned to one of the treatments T \ 
(standard drug; control), say, and the remaining member of 
the pair is assigned to the second treatment T 2 (new drug; 
case) say, and the various c possible responses are recorded 
for each subject. If in particular the responses of each matched 
pair of subjects are classified into c = 3 mutually exclusive 
categories or classes, the data presentation format is as in 
Table 1 below. 

Table 1: Format For Presentation Of Data On ‘C’ = 3 


Outcomes in a Clinical Trial of Matched Pairs 



Outcome Category for Control (Standard T 7 ) 

Outcome 
Category for 
Cases 

(Experimental 
Condition T 2 ) 

1 

2 

3 

Total Ok.) 






1 

nn 

n 12 

n 13 

n i. 

2 

n 21 

n 22 

n 23 

n 2. 

3 

n 31 

n 32 

n 33 

n 3. 

Total (n.j) 

n.i 

n. 2 

n.3 

n..(=n) 


Each entry in Table 1 consists of a matched pair of case and 
control subjects. For example n^ is the number of pairs in 
which the case is in category T response while the 
corresponding control subject is in outcome or response 

category j for i, j = 1,2,3. riband are respectively 

the total number of pairs in which the case is in category 4 i’ 
response and the control is in category ‘j ’ response 
for Lj ~ 1,2,3 . 

In all, there are a total of 

n — n. . — — 2_ ? -=i j 

pairs of subjects studied. A null hypothesis usually tested 
using the Stuart-Maxwell test is that case and control subjects 
or patients do not differ in their response to the treatments. 
The corresponding star-Maxwell test statistic for this purpose 
is 



H ng.tx^ 




2 + n 


1 «n 

— JU 


n 


ng. EH 13 




11 


} 



which under H a has approximately the chi-square 
distribution with 2 degrees of freedom, for sufficiently large 
n = n. . where 
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and 



n ij * H jl 



for i, j = 1,2,3; i == j 

Now to develop the proposed method let, as in 
Stuart-Max well method, the difference between the number 
of pairs of respondents in the i th category of responses for case 
and j th category of responses for control (Miettinen, 1969; 
Maxwell, 1970; Everitt, 1977; Stuart, 1955) be d t (equation 2) 


which is independent of Ttj j , i = 1, 2, 3 the number of pairs in 
which both case and control subjects have the same response 
or outcome. Also let 


d ij = n ij - n ji • ■ • (4) 

which is the difference between the number of pairs in 
which the case is in the response category i and the control is 
in the response category j and the number of pairs in which 
the case is in response category j and the control is in the 
response category i; ij = 1, 2, 3 i ^ j. 

Now having selected our random sample of n matched pairs, 
let xu be the response by a member of the randomly selected 
i’th pair of patients or subjects randomly assigned treatment 73 
(control, standard drug) and x i2 be the response by the other 
member of the pair of patients or subjects assigned treatment 

T 2 (case, new drug) for i = 1,2, , n. We here assume for 

ease of presentation, but without loss of generality, that the 
three mutually exclusive possible response categories have 
been ordered from the highest or most serious (lowest or least 
serious) level of response to the lowest or least serious 
(highest or most serious) level of response. For example, a 
patient’s response to a treatment for an illness or disease may 
range from recovered, through no change to dead; a 
subject’s response to a screening test may range variously 
from definitely positive, no change, to definitely negative. A 
candidate’s or student’s performance in a job interview or 
examination may range from very poor, good, to excellent. 


We here assume that those responses have been appropriately 
arranged either in increasing or decreasing order of 
seriousness. 

Now let 



l,Ifx/ 2 , i.e the response by the member in the i th 
pair of patients or subjects assigned treatment T 2 
(case) is a higher or more serious (lower or less 
serious) level of response than x ih the response 
by the other member of the pair assigned 
treatment 73 (control ) for all the 3 response 
categories. 


0, if xu and x i2 , are the same level of response for 
the two patients or subjects in the i th pair for all 
the 3 response categories 

-1, if xu, the response by the member in the i th 
pair of patients or subjects assigned treatment T 2 
(case) is a lower or less serious (higher or more 
serious) level of response than x u , the response 
by the other member of the pair assigned 
treatment 73 (control) for all the 3 response 
categories 
For i =1, 2,...,n 
This means that u { assumes the value 1 , if the response of the 
member of the i th pair of patients administered treatment T 2 
(case) is a higher or more serious (lower or less serious) level 
or response than the response of the other member of the 
pair administered treatment T t (control); 0, if the response of 
the two members of the pair are the same, and -1, if treatment 
73 (case) the response of the pair administered is a lower or 
less serious (higher or more serious) level of response than 
the response of the other member of the pair administered 
treatment 73 (control) for all the 3 response categories. 

Now let 



K 


+ 


P{Ui = 1); JT° = P(u t = 0); n = p(u t - -1) 

... 6 


Where 

7T + + R° + 7T _ = 1 ... 7 

Fet 

w= 2"=lU, ...8 

Now 

E(u i ) = 7T + -n~ ...9 

And 

_i_ _|_ ^ 

Var(u i ) = 7z + n -{tz — tz ) ...io 

Also 

E(W) = Y! i=l =n(7z + -7z~) ...li 


Note that 7T 7T is the differential response rate between the sub -populations administered treatments 73 (case) and 73 
(control) respectively in the paired population of patients or subjects for all the c = 3 response categories and is estimated by 



71 
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Note also that 7T + p 7T 1 " 1 and 7T which are respectively the probabilities that a randomly selected case is at a higher (or more 
serious) level, the same or lower (or less serious) level of response than the corresponding control subject in the pair for all the 
three response categories are estimated using the frequencies in Table 1 and following the specification in Equation 5 as 


7 T- =P + = — = — ^ 


.+ 


/ + 


y? y? - - 


n 12 +n 13 +n 23 


n n n 

0 f° 2f =1 riii «ii+n 2 2+n 3 3 


ft 0 - p° = J — = 


... 13 


14 


■n 


71 


71 


And 


n 


_ p- _ f_ _ 1 >j _ ™2± rw ai r »3 2 


y? y ? n - 


71 


n 


Tlryi +n a H +tl. 


... 15 


Where /V 0 and f are respectively the number of l’s, 0’s and -l’s in the frequency distribution of the n values of these 

numbers in u in accordance with Equation 5, i= 1,2 n 

Hence using these results in equation 12, we have that 

W = f + — f- = — 2 t=a E ] ? =i n ii = (n lz — n 21 ) + (n 13 — n 31 ) + (n 23 — « 32 ) 

i<? i>j 


...16 


Now from equations 8 and 10 we have that 


Var(W) = 'Z?= 1 Var(u i ) = n((n +n )- 


17 


Whose sample estimate is from equations 13 and 15 as 


Var(W) = n((n* + n ) — (tt 4 ' — ft ) 2 ) ... 18 


-fr 


As noted above 7T is the proportion of pairs of case and 
control subjects in which on the average the response rate by 
the sub-population of patients or subjects administered 
treatment T 2 (experimental, case) is greater (less) than the rate 
by the sub-population of patients or subjects administered 

treatment Tj (standard, control); while 71 is the proportion 
of pairs in which on the average the response rate by the 
sub-population of patients or subjects administered treatment 
T l (standard, control) is greater (less) than the response rate 
by the sub-population of patients administered treatment T 2 
(experimental, case) in the paired population of patients or 
subjects for all the three response categories. Hence the null 
hypothesis that there exists no difference between the 
response rates by the sub-population of patients administered 
treatment T 2 (experimental, case) and the sub-population of 
patients administered treatment (standard, control) in the 
paired population of patients for all response categories is 
equivalent to the null hypothesis 

tz~ — tt~ ~ 0 versus H ± : n~ —n~ == 0 

... 19 

To test this null hypothesis, we may use the test statistic 
Note that 


x 2 = 


w j 


tv 


Var(W) 

c/ + -/-y 


«C?r + -lr'?r — (rr + -trrt' 


r*r- 


_ t r-ry 




... 20 

Which under has approximately a chi-square distribution 
with 1 degree of freedom for sufficiently large n. Although 
strictly speaking, the test statistic in Equation 20 has a 
Chi-square distribution with 1 degree of freedom, however 
because its construction in equation 5 involves a combination 
of some c = 3 response categories, to help increase its power 
and reduce the chances of erroneously accepting a false null 
hypothesis (Type II error), it is here recommended that all 
comparisons should be made against critical Chi-square 
values with 3 - 1 =2 degrees of freedom, instead of 1 degree of 

freedom. Hence here Hq is rejected at the a level of 
significance if 

* 2 > ^Cl-^z) ...21 

Otherwise is accepted. 


+ n “) = + £f =2 E£ =1 %- = n((n 12 + n 13 + n 23 ) + (n 21 + n 31 + n 32 )) 


...22 


Hence using equation 16 and 22 in equation 20 the test statistic becomes 


X 2 = 


■ 

f Cn 1 2 — n 2 1 -■ + i a g i J + (^2 g —r a 2 )) 


n 


12 


+71 ± a +7ln n +rcrn +71 n ^ +71 


22 


21 


a l 


a 2 


-( 


( n 12 — 'ft 2i) ■+ (fti a - ft a i)+ ( fta a “ft 32) 


ft 


) 


... 23 
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If there are only two possible outcomes or responses, that is c = 2, equation 20 under H 0 reduces to a modified version of the 
McNemar test statistic which is 

Gi 12 — tz 2 .i J 2 


X 2 = 


n ±z +n zl - 


fa 12 2i> 2 

n 


24 


This has a chi-square distribution with 1 degree of freedom. 

Note that equation 24 has smaller variance than the usual McNemar test because of its modification to provide for possible ties 
between case and control subject pairs in their responses. 

If c = 3, equation 20, under H 0 , reduces to 


X 2 = 


i 

((n 12 -n 21 )+(ri la -n al ’J+fa 2a -n a2 )) 


™12+™ia +™21+«2a +"31+"32 


Cfal2-^21> + fal 3 3 1) + fa 2 3 -■« 3 2>) ' 


n 


And this has a chi-square distribution with c-l=3-l=2 
degrees of freedom. 

Finally, note that if we let 

m if - n tj + n Ji , i,j = 1, 2, 3 ; i = j 

...26 

Then the test statistic of equation 20 can be written in an 
easier and more compact form using equations 4 and 26 as 


U = 


... 27 


I- I* 

"1=1"/=! 


m L/“ 


(zf =1 ^ =1 d i; y 


Yl 


25 


And from Equation 15, we have that 

16+3+4 23 = 0.152 


— 


n = 


151 151 

1} 


Note that St u = 1 - (0.271 + 0.152) = 0.577 
Also from Equation 1 1 , we have that 
E(W) = n(it+ - tt“) = 151(0.271 + 0.152) = 
151(0.119) = 17.969 

From Equation 17, we have that 

Var(W ) - 1S1[(0.271 + 0.152) - 

(0.271 - 0.1S2) 2 ] = 61.735 


If equation 20 leads to a rejection of the null hypothesis of 
equal response rates then one may wish to proceed to identify 
the response categories or combination of categories that may 
have led to the rejection of H 0 . This is done by appropriately 
pooling or combining the response options into (2) groups 
and apply the McNemar test (McNemar 1983, Somes 1983, 
Sheskin 2000) to each of the groups. In all cases comparisons 
are made using critical chi-square values with 2 degrees of 
freedom to again avoid erroneous conclusions. 

A. Illustrative Example 1 

We here use data on matched pairs of 151 patients from a 
controlled comparative clinical trial who manifest three 
possible responses to illustrate the proposed method. Suppose 
the data in Table 2 are obtained by assigning a standard 
treatment T ) (control) and a new treatment T \ (case) at random 
to members of each pair of a random sample of 151 pairs of 
HIV patients matched on age, gender and body weight used in 
a controlled clinical trial to compare the effectiveness of two 
HIV drugs. 


Table 2: Data from Controlled Comparative Clinical Trial Using Matched 
Pairs with Three Responses 



Standard Treatment 7) (control) 

New reatment 
T 2 (case) 

Improved 

No 

Change 

Dead 

Total Ok.) 

Improved 

60 

31 

4 

95 (=n L ) 

No Change 

16 

24 

6 

46 (=n 2 .) 

Dead 

3 

4 

3 

10 (=n 3 .) 

Total (n j) 

79 

(n.i) 

59 

(n. 2 ) 

13 

fa. 3) 

151 (=n ) 


To test the null hypothesis that case and control do not differ 
in their response to the treatments (Equation 19,) we have 
from equation 13 that 



31+416 

151 


41 

151 


0.271 


Hence from Equation 23, we have that 

1(31-1 6} 4 (4- 3} 4 ( 6-4}) 


* 2 = 


324 

63.5S6 


31444 64164344 


-( 


(ai-ie) + Ca - + (-&-40 


151 


)' 


= 5.064 (p — value = 0.008 


Which with c-l=3-l=2 degrees of freedom is highly 
statistically significant at a = 0.01. 

We may therefore conclude at the 1 percent significance level 
that the treatments have differential effects on the patients. 

If we had used the Stuart/Maxwell method to analyse the data 
we would have from Equation 2 that 

d i = 95 - 79 = 16, d 2 = 46 - 59 = -13, a T s = 
10 - 13 = -3 


Also 


71, i 4n 


letting 


n i; — ~ i iL .■ £ — lj'2, 3 ; / — 1,2,3; i === j f we have 

31416 47 _ _ _ _ 443 7 

n 23.5; Tl-Jj n 

2 2 1 1 d 2 2 

644 _ 10 _ 

2 2 


3.5; n 23 = 


Hence using the Stuart Maxwell test, we have 
2 _ 5(16} = 43 5( — 13} = 423.5( — 3} = _ 1B94 

2 ( ( 2 3 . 5} ( 3, 5} 4 ( 2 3. 5} ( 5} 4 ( 3 &)£&)) 434.5 

4.359 (p — value = 0.12) 


Which, with 2 degrees of freedom, is statistically significant 
at the 2 percent level of significance but not statistically 
significant at the 1 percent level of significance, the usually 
used norm in medical research? 

Thus the present (extended) method leads to a rejection of the 
null hypothesis H 0 while the Stuart/Maxwell test statistic 
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leads to an acceptance of the null hypothesis at the 1 percent 
significance level. Hence the Stuart/Maxwell Test is likely to 
lead to an acceptance of a false null hypothesis (Type II 
error). This means that the present test statistic is likely to be 
more efficient and more powerful than the Stuart/Maxwell 
test statistic. 

As noted above, the present method may also be used to 
analyse quantitative or numeric data obtained in matched 
controlled studies. Often, responses from controlled 
experiments are reported as numeric scores assuming all 
possible values on the real line such that scores in the interval 
(ci, c 2 ) where ci and c 2 are real. For example, these responses 
may be values on the real line such that scores in the interval 
(ci, c 2 ) where ci and c 2 are real numbers (ci < c 2 ), indicate that 
the responses by the subject concerned are normal, negative, 
condition absent, etc; values less than ci indicate that the 
subjects have abnormally low scores; and values above c 2 
indicate that the subjects have abnormally high scores. It is 
also possible to have situations in which subjects have scores 
that are either some c 3 units below Ci or some c 4 units above 
c 2 . These subjects may be concerned to have non specific or 
non definitive manifestations. Subjects whose scores are 
below c 3 and above c 4 may be considered to have critically 
abnormal manifestations, one below the critical minimum and 
the other above the critical maximum normal scores. If these 
results are considered important manifestations, then the first 
set of subjects may be grouped into three response categories, 
while the second set of subjects may be grouped into five 
response categories for policy and management purposes. 

To illustrate the use of the present method when the case and 
control subjects in matched controlled studies have 
quantitative scores with three possible outcomes for instance, 
we would proceed as follows: 

Suppose as above, a random sample of n pairs of case and 
control subjects are used in a controlled experiment on two 
procedures T 7 (control, standard) and T 2 (case, experimental 
procedure). Suppose as before, one member of each pair is 
randomly assigned treatment T 7 (control, standard) and the 
remaining member assigned treatment T 2 (case, experimental 
procedure). 

Let y u and y i2 be respectively the responses or scores with real 
values, quantitatively measured, by the subjects assigned 
treatment T 1 (control) and T 2 (case) for the i th pair of subject 
for 2 2-j ... j Tt-i 

Then u t of Equation 3 may now be defined as 
u i = 

■" 1, if either y i2 < c 1 and c 1 < y tl < c 2 or y i2 < 
and y fl > c 2 or q <y i2 < c 2 and y a > 


< 


in the (i , j ) th case - control response classification for 
2 — 1,2,3 and J — 1j2j3. 

Specifications similar to Equation 28 can also be easily 
developed for more than three quantitative response 
categories if of interest. Now to use Equation 20 to analyse 

these data, we would again simply define 7F , 7T 7 T and 
W as in Equations 6-8. Then data analysis proceeds as usual. 


B. Illustrative Example 2 

A medical researcher is interested in knowing the relationship 
between heart disease and low density Lipo -Protein Levels 
(LPL). Using a random sample of 36 non-heart disease 
patients and another random sample of 36 heart disease 
patients, she paired each non heart disease patient with a heart 
disease patient matched in age, gender, body weight and 
occupation and then measured the LPL of each subject in the 
pair. The results are presented in Table 3 

Table 3: LPL levels of Paired Samples of Patients in a 
Clinical Trial 


0, if either y i2 < c ± and y i± < c 1 or c 1 < y i2 < 
and c 1 < y tl < c 2 or y i2 > c 2 and > 7 n > C 2 

— 1, if either c 1 < y i2 < c 2 and y i± < c t or y i2 > 

and y tl < c 1 ar y i2 > c 2 and c 1 < y i± < c 2 

...28 

For 2 = 1.. 2, ... j 71. 

Note that this specification may be depicted in a 3 x 3 table if 
we let n t j be the number of paired case and control subjects 




S/N 

Paired LPL levels 

Scores (u t ) 

1 

(1.97,4.14) 

0 

2 

(3.70,1.57) 

-1 

3 

(5.40,5.60) 

0 

4 

(2.60,5.10) 

1 

5 

(3.10,1.50) 

-1 

6 

(1.48,4.56) 

1 

7 

(1.69,1.70) 

0 

8 

(4.97,1.21) 

-1 

9 

(2.34,2.51) 

0 

10 

(3.95,1.55) 

-1 

11 

(4.84,1.25) 

-1 

12 

(4.65,4.59) 

0 

13 

(1.29,1.37) 

0 

14 

(1.15,6.24) 

1 

15 

(5.41,1.20) 

-1 

16 

(4.62,1.25) 

-1 

17 

(2.02,1.53) 

-1 

18 

(1.45,1.30) 

0 

19 

(5.31,L07) 

-1 

20 

(5.18,4.37) 

-1 

21 

(4.52,5.38) 

1 

22 

(5.03,3.34) 

-1 

23 

(5.21,4.55) 

0 

24 

(4.74,5.59) 

0 

25 

(3.76,3.96) 

0 

26 

(5.21,3.50) 

-1 

27 

(5.09,4.66) 

0 

28 

(1.97,4.14) 

0 

29 

(2.60,5.10) 

1 

30 

(1.69,1.70) 

0 

31 

(3.95,1.55) 

-1 

32 

(1.29,1.37) 

0 

33 

(4.62,1.25) 

-1 

34 

(5.31,1.07) 

-1 

35 

(5.03,3.34) 

-1 

36 

(3.76,3.96) 

0 


LPL Normal range (1.68, 4.53) 
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Applying the specification of Equation 28 to the LPL levels of 
Table 3 with ci = 1.68, the lowest and 4.53 the highest normal 
values respectively we obtain the corresponding scores u iof l s , 
Cb and -l s shown in the 3rd column of this table. 

Thus we ha ve/ + = 5 ,f°= 15 and / = 16. Hence, we have 

from Equations 12-15 that 


The null hypothesis to be tested is that heart disease patients 
and non-heart disease patients do not differ in their LPL 
which is equivalent to testing 

H ( j : n ~ — ft ~ = 0 versus H . L : n~ — n~ =p 0 

Using the test statistic of equation 20 or 23, we have that 


* = — = 0.139; = — = 0.417 and IT - 


n 

36 

— = 0.444 

36 


36 


From Equation 1 8, we have that the estimated variance of W is 


Var(W ) = 36(0.139+ 0.444) - 
20.988 — 3.361 = 17.627 


(-ii) 

36 


2 C-llP 

X — 

17.627 

0.0391) 


121 


17.627 


= 6.864 (p value = 


which with c-l = 3- l= 2 degrees of freedom is statistically 

significant at the 5 percent level 2 = S.99l) 

We may therefore conclude that heart disease patients and 
non- heart disease patients do infact differ in their LPL. 

The data of Table 2 may infact be represented by a 3 x 3 table 
and following the specifications of Equation 28 with Cj = 1.68 


and c 2 = 4.53 to aid in clearer analysis as in Table 3. 


Table 4: Distribution of Scores u t of Matched pairs of case and control subjects of Table 3 



Control (Tj) Scores 

Case(T 2 ) Scores 

Below Normal 
(y u <1.68) 

Normal (1.68<y, 7 < 
.53) 

Above Normal (y u > 
4.53) 

Total 

Below Normal (y i2 < 1.68) 

4 

0 

2 

6 

Normal (1.68 < y i2 < 4.53) 

5 

6 

3 

14 

Above Normal 
(j/2> 4.53) 

7 

4 

5 

16 

Total 

16 

10 

10 

36 


To re-analyse these data consistent with the generalized 
method, we have from Equation 1 3 that 

= — = 0.139 

36 36 

From Equation 14, we have that 

ft * = = — = 0.417 

36 36 

And from Equation 15, we have that 

51714 16 _ 

n = = — = 0.444 

36 36 


These are the same results obtained earlier using the scores in 
Table 3. We would therefore obtain the same values of W 
(-11) and chi-square (6.864) and arrive at the same 
conclusions. Hence, the present example illustrates how to 
analyse matched quantitative test scores without first 
converting them into frequency data. 

The data of Example 2 as presented in Table 4 may also be 
analysed using the Stuart-Max well test. However as already 
pointed out, the Stuart/Maxwell test statistic is almost as 
powerful as the test statistic used in the proposed method 
presented here when the two methods are used with data of 
equal sample sizes 


III. Summary and Conclusion 

We have in this paper presented and discussed a generalisable 
statistical method for the analysis of three responses or 
outcomes in case - control studies, including situations in 
which the data being analysed are either quantitative or 
qualitative frequency data. 


Test statistics are developed for testing the statistical 
significance of differences between responses. 

The proposed methods are illustrated with sample data and 
shown to be more powerful than the usual Stuart/Maxwell test 
when the two methods are equally applicable to a set of data. 
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